Repository navigation
feat(streaming): add opt-in per-key stream cost breaker - #66
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
multillm-proxy | 470a092 | Oct 09 2026, 05:53 AM |
Summary
Adds an opt-in per-key stream cost breaker. When a key has a cap, the gateway tracks a stream's running cost and stops it with the protocol's real error before it emits a chunk that would cross the cap. With
STREAM_COST_BREAKER_ENABLEDunset oroff(the default), or a key's cap left null, streams, account queries, writes, key serialization and storage are byte-for-byte unchanged. Every new hook checks the flag first and falls through to the existing code.STREAM_COST_BREAKER_ENABLED: empty means off; a malformed value is off with one warning that leaves the value out.max_stream_cost_usdis set through the existing administrator key controls and stored as integer micro-USD. It must be non-negative, finite, at most 1,000,000 USD and exact to the micro-USD. Zero is valid.MODEL_PRICING_USD_PER_MILLIONinput, cache-read, cache-write and output buckets.stream_cost_unpricedbefore handoff. An input estimate already over the cap returns 429stream_cost_cap_exceeded.measured. Otherwise aconservative_estimatecounts each emitted UTF-8 byte as an output token, which deliberately overestimates.data: {"error": {"code": "stream_cost_cap_exceeded", …}}, Anthropicevent: error, or Responsesresponse.failed.finish_reason: stop, a success body or a successful[DONE].Details are in
docs/stream-cost-breaker.md.Turning it on
Apply migration
0030before setting the flag. With the flag on and the column missing, D1 account reads and writes return 503stream_cost_storage_unavailablerather than dropping controls or falling back. In practice, that means key authentication fails until the migration is applied.Wiring
services/auth_service.pygets small call sites only. Account reads and writes, local SQLite schema setup, cached user rows and administrator updates pass throughstream_cost_breakerhelpers, which return the existing results while the flag is off.services/key_controls.pyacceptsmax_stream_cost_usdonly while the flag is on, and validates it, stores it and shows it in the public key record.services/request_accounting.pyprepares the breaker for capped streaming requests, wraps event streams and records stopped streams with their hold.services/upstream_transport.pybinds the upstream cancellation so the breaker can close it.worker/control-users-d1.mjsreads and writes the cap column only while the flag is on.worker/stream-cost-breaker.mjsprovides the same breaker for native Worker streams. It is connected to the native generation path with the Wave 6 integration.Shared registrations
tests/test_stream_cost_breaker_worker.mjsintest:worker.worker/stream-cost-breaker.mjsin the edge module inventory.STREAM_COST_BREAKER_ENABLEDin the Container env keys.0030_stream_cost_caps.sqladds the nullablecontrol_users.max_stream_cost_microusdcolumn (ALTER TABLE … ADD COLUMN). No new tables.Stacked on #62
Migration numbers must be contiguous in each PR, so this branch merges
feat/realtime(#62, migration 0029). Merge #62 first. The learned cooldown and usage receipt PRs stack on this one.Tests
tests/test_stream_cost_breaker.pyandtests/test_stream_cost_breaker_worker.mjscover:npm run test:worker: 1,667 passed.npm run test:integration: 7 passed.cf:migrations:checkreports 30 contiguous, additive migrations.test:deploy, free routes, client headers and thewrangler deploy --dry-runbundle all pass.tests/test_frontend_build.mjsfails locally because@tailwindcss/cliis not installed. It fails the same way on main.Test fixture fixes
Two existing tests build a partly migrated D1 database by skipping
0003_control_users.sqltogether with every migration that alters that table. Migration0030also alterscontrol_users, so both now skip it as well:tests/test_d1_schema_worker.mjs(the/readyschema test) andtests/test_d1_integration.mjs(the unmigrated-accounts integration test).Apply the migration before enabling the flag:
npx wrangler d1 migrations apply multillm-intelligence --remote.CI notes
python/nodejobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.AdmissionCoordinatorDurable Object migration, whichwrangler versions uploadcannot apply. This is expected untilwrangler deployruns on main.This is part of Wave 6. The README line comes with the Wave 6 integration PR.